Virtual model viewing methods and apparatus

ABSTRACT

Virtual model viewing methods and apparatus are described. According to one aspect, a virtual model viewing method includes using a user device, accessing model data which may be used to generate a plurality of views of a virtual model, using a camera of the user device, generating image data of content of the physical world, and using the model data and the image data, creating one of the views of the virtual model which comprises at least some of the content of the physical world, and displaying the one view of the virtual model using a display screen of the user device.

TECHNICAL FIELD

This disclosure relates to virtual model viewing methods and apparatus.

BACKGROUND OF THE DISCLOSURE

Three-dimensional computer graphics technology is often used inarchitectural design applications, mechanical design applications, andgames to allow a user to create, view, and navigate a virtual scene.More recently, the sophistication and capabilities of mobile computingdevices have increased significantly and many devices are capable ofexecuting applications which were previously executed on desktopmachines such as personal computers.

The present disclosure is directed towards methods and apparatus toenable a user to navigate virtual models, such as virtual models ofarchitectural structures, and additional methods and apparatus relatedthereto which are described in detail below according to exampleembodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments of the disclosure are described below with referenceto the following accompanying drawings.

FIG. 1 is an illustrative representation of a view of a virtual scenegenerated by a user device according to one embodiment.

FIG. 1A is an illustrative representation of another view of a virtualscene generated by a user device according to one embodiment.

FIG. 2 is a functional block diagram of a user device according to oneembodiment.

FIG. 3 is an illustrative representation of a camera and correspondingfield of view according to one embodiment.

FIG. 4 is an illustrative representation of a view of a virtual scenegenerated by a user device according to one embodiment.

FIG. 5 is an illustrative representation of movements of a user and auser device according to one embodiment.

FIG. 6 is a flow chart of a method of updating an orientation of avirtual camera according to one embodiment.

FIG. 7 is a flow chart of a method of detecting a footstep and updatinglocation of a virtual camera according to one embodiment.

FIG. 8 is a graphical representation of raw and filtered accelerationdata obtained by a user device according to one embodiment.

FIG. 9 is a graphical representation of filtered acceleration dataobtained by a user device according to one embodiment.

FIGS. 10 and 11 are graphical representation of different attenuationfunctions according to one embodiment.

FIG. 12 is a perspective view of a virtual scene of a virtual model anda backdrop according to one embodiment.

FIG. 13 is an illustrative representation of a view of a virtual sceneincluding content of the physical world according to one embodiment.

DETAILED DESCRIPTION OF THE DISCLOSURE

This disclosure is submitted in furtherance of the constitutionalpurposes of the U.S. Patent Laws “to promote the progress of science anduseful arts” (Article 1, Section 8).

The reader is directed to the following applications which were filedthe same day as the present application: “Virtual Model NavigationMethods and Apparatus”, naming Greg Wells, Chris Tenbrink, Carl Wielandand Dan DeBroeck as inventors, and having attorney docket numberAD23-008; “Virtual Model Navigation Methods and Apparatus”, naming GregWells, Chris Tenbrink and Carl Wieland as inventors, and having attorneydocket number AD23-010; and “Step Detection Methods and Apparatus”,naming Greg Wells, Chris Tenbrink and Carl Wieland as inventors, andhaving attorney docket number AD23-011, and teachings of which areincorporated herein by reference.

In some embodiments, a virtual model viewing method comprises using auser device, accessing model data which may be used to generate aplurality of views of a virtual model, using a camera of the userdevice, generating image data of content of the physical world, andusing the model data and the image data, creating one of the views ofthe virtual model which comprises at least some of the content of thephysical world, and displaying the one view of the virtual model using adisplay screen of the user device.

Some of the aspects of the disclosure are directed towards method andapparatus for generating and navigating virtual models, such asthree-dimensional (3D) virtual models, which may be displayed andinteracted with using a computing system. At least some embodimentsdescribed below are directed towards generating and navigating virtualmodels of architectural structures (e.g., houses) although the describedembodiments may be utilized to generate and interact with other virtualmodels in other examples, and perhaps having vastly different scales(e.g., virtual models of the human circulatory system, an internalcombustion engine, etc.).

As described further below, some illustrative aspects of the disclosureare directed towards utilization of data from sensors of a user device,which detect movements (translations and rotations) of the user device,to navigate a plurality of different views of a virtual model which aredisplayed using the user device. Example user devices include mobilecomputing devices, such as tablet computers, cellular telephones,smartphones, laptop computers, eyewear computing devices (e.g., GoogleGlass™, etc.) as well as other computing devices, such as desktoppersonal computers or workstations.

Additional aspects of the disclosure are directed towards combiningcontent of the physical (real) world with virtual content of the virtualmodel as described in example embodiments below. The discussion proceedsbelow with respect to describing example systems and methods to generatedifferent views of virtual models during user navigation of the virtualmodel using an example computing device.

In one embodiment, three-dimensional computer graphics technology isutilized to allow a user to create, view, and navigate a virtual model.A virtual scene includes information regarding the virtual model and avirtual camera which is used to generate views of the virtual modelwhich are displayed for the user. The virtual scene contains informationutilized by the computer graphics technology to produce an image on thescreen that represents the virtual model stored in computer memory. Forexample, the virtual scene may include information about geometricobjects, their materials, and lights used to illuminate them.

In one implementation, an architectural computer-aided design (CAD)system is used to design and/or modify a virtual model, such as avirtual model of an architectural structure (e.g., Chief Architect andRoom Planner products available from Chief Architect, Inc.), and someembodiments of the disclosure enable users to navigate the virtual modelincluding controlling the generation of desired views of the virtualmodel. In some embodiments disclosed herein, a graphics engine, such asUnity available from Unity Technologies, San Francisco, Calif. (e.g.,www.unity3d.com), is used. Different methods or systems may be used tonavigate virtual models and create different views of the virtual modelin other embodiments.

In one example, the virtual scene is composed of a set of objects andeach object includes a set of components that specify how the objectappears in the virtual scene. For example, an object may include atransform which defines the object's position, orientation, and scalewith respect to some parent transform. In the most basic case, anobject's transform has no parent and the object's transform defines theobject's position, orientation, and scale, with respect to a virtualscene's global origin. In a more complicated case, an object's transformmay have some other transform as its parent which means that theobject's position, orientation, and scale are a function of the parenttransform. In this way, a hierarchy of transforms can be created andeach transform in the hierarchy gets evaluated to determine the finalposition, orientation and scale of the object in a given view of thevirtual model.

In one illustrative example, a virtual model of a bicycle may be createdwhere the bicycle frame is an object that contains transform T1 andcoordinates for all parts of the frame are given with respect totransform T1. The wheels are also objects containing transforms T2 andT3, and coordinates for the wheel geometry are given with respect tothese two transforms. The wheels are positioned and oriented withrespect to the frame by defining their offset and orientation withrespect to their parent transform, T1. The whole assembly is positionedand oriented in the scene by specifying the position and orientation ofT1, which has no parent or may have a scene level parent.

Objects can contain other components besides transforms. For example, amesh is a component that defines the 3D geometry of the object using aset of 3D vertices and a set of connections between the vertices.Usually, the connections between vertices produce a set of triangleswhich are used to represent the surfaces of objects that are intended tobe shown in the virtual scene.

Referring to FIG. 1, a computing device or system, which may be utilizedas a user device 10, is shown displaying an example view 12 of aninterior room of a virtual model in the form of an architecturalstructure. Example computing devices or systems (e.g., personalcomputers) are configured to display images for observation by a userand include processing circuitry configured to control the generationand display of the images. The computing devices or systems may beportable in some arrangements, such as the mobile computing devicesdiscussed above.

The user device 10 includes a display screen 14 which may be used togenerate visual images for viewing by the user. The display screen 14may also be interactive to function as a user interface (e.g., graphicaluser interface) which is configured to receive user inputs via a usertouching the display screen 14 in some embodiments. In the example ofFIG. 1, the display screen 14 depicts an image in the form of a window16 which corresponds to an application being executed by the user device10 and which includes the view 12 of the virtual model in a respectivevideo frame.

In one embodiment, each wall, floor, ceiling, window, door, andfurniture piece (and other items) of a virtual model of an architecturalstructure is an object 18 containing a mesh (only some of these objectsare shown in FIG. 1). In some embodiments, a scene coordinate referenceframe or system 19 of the virtual model is at least partially aligned(aligned in x, y plane) with a global coordinate reference frame orsystem of the physical world such that the foundation of thearchitectural structure is shown below the roof in views of the virtualmodel. In some embodiments, locations of the virtual model correspond tolocations in the physical world. The floor object 18 of thearchitectural structure may be referred to as a base plane and which maybe defined in a x, y plane (i.e., the footprint of the structure) andthe walls of the structure may extend upwards along the z axis from thex, y plane. In some embodiments, the scene and global coordinatereference frames are completely aligned (e.g., a North direction of thearchitectural structure is aligned with a North direction of thephysical world).

As discussed below in some embodiments, the user device 10 has arespective device coordinate reference frame or system 21 andtranslations of the user device 10 are monitored relative to the devicecoordinate reference frame and rotations of the user device 10 aremonitored relative to the global coordinate reference frame in oneembodiment. The scene coordinate reference frame 19 is left-handed andthe device coordinate reference frame 21 is right-handed in oneembodiment.

In addition to use of 3D coordinates of the scene coordinate referenceframe to define its position in virtual space, each vertex in the meshalso has a set of three numbers that define a normal vector for thevertex in one embodiment. The normal vector represents the normal of thesurface which the mesh is approximating at the location of the vertex inone embodiment.

Also, in one embodiment, each vertex can contain a set of texturecoordinates so when a 2D image is attached to the object, the texturecoordinates define a location on the 2D image that should be mapped tothe vertex location when the mesh is rendered. In this way, an image maybe applied to a mesh to add graphical details that would be difficult tomodel geometrically with portions of the mesh. Each vertex in the meshcan also contain position coordinates and a normal vector.

One example method for implementing navigation of virtual models isdiscussed below with use of the Unity graphics engine. Morespecifically, the Unity graphics engine provides an object for a virtualcamera through which a user views the virtual model. For example,different views of the virtual model may be drawn in frames using thevirtual camera which are snapshots of the virtual scene at respectivemoments in time. The virtual camera is considered to be within thevirtual scene of the virtual model in one embodiment. The virtual cameracontains a transform to define its view point (position) and viewdirection (orientation) with respect to the rest of the scene in thescene coordinate reference frame. The view point position may be definedin three coordinates in the scene coordinate reference frame and theview direction may be defined as an orientation in the scene coordinatereference frame. The views of the virtual model change as one or both ofthe view point and view direction of the virtual camera change. The viewpoint location and/or orientation of a virtual camera within the virtualscene may be changed as a result of inputs and these changes result inchanges of the view of the virtual model. Example user inputs includeinputs entered via user controls of the user device including thegraphical user interface (or other appropriate interface such as a mouseor keyboard of the user device) and/or inputs generated by movements(e.g., rotations and translations) of the user device as describedfurther in example embodiments below.

As described in some example embodiments herein, user inputs receivedvia a user control (e.g., joystick), inputs from location sensors,and/or inputs resulting from translations and rotations of the userdevice may result in the change of the view point and/or view directionof the virtual camera resulting in different views 12 of the virtualmodel. As shown in FIGS. 1 and 1A, different portions of the virtualmodel are shown in different views 12, 12 a resulting from a change inthe view point and/or view direction of the virtual camera.

By convention in Unity, the virtual camera is oriented so that the useris looking along the positive z axis of the transform with the positivey axis of the transform defining the ‘up’ direction. The virtualcamera's position and/or orientation in the scene can be changed bymodifying its transform. The virtual camera also has other parametersthat effect how the scene is displayed in views (e.g., Field of View,near and far clipping plane distances, whether the projection isorthographic or perspective).

The Unity graphics engine performs an initial setup of a virtual scene,such as for a house, upon start-up. This consists of instantiating anobject for the various existing elements of a plan of the house whichhave been stored in a database during a previous session or otherwiseaccessed. For example, an object with a mesh is created for each wall inthe virtual model of the home. In the same way, objects for eachfurniture item, window and door in the virtual model are instantiated.Additionally, an object for the virtual camera is created and itstransform is modified to place it at some predefined position (i.e.,view point within the virtual scene) and orientation (i.e., viewdirection within the virtual scene) with respect to the other objects inthe virtual scene.

Once all of the objects have been setup, the Unity graphics engineperforms a series of calculations that uses information stored with allof the objects in the virtual scene to determine what color each pixelon the display screen should be set to. The Unity graphics engine drawsa picture of the virtual scene as a view of the virtual model on thedisplay screen at a particular instant as a frame, for example as avideo frame.

A single update of the display screen is referred to as a frame update.User device 10 can perform these calculations sufficiently fast thatframes of the display screen can be redrawn at a plurality of differentmoments in time (e.g., a frame rate between 15 and 150 times per secondin one example), which is sufficient to produce the illusion of movementin the virtual scene typical of video.

The Unity graphics engine allows scripts to be connected as componentsto objects. This provides a method for programmatically defining thebehavior of objects. On each frame update, the Unity graphics engineiterates over all of the objects in the view to be drawn, looking forobjects with an attached script. When it finds an attached script, theUnity graphics engine invokes the ‘Update’ method defined in the script(if the method exists). The ‘Update’ method can be defined by theapplication developer to query and modify data in the application toproduce a particular behavior. For example, the ‘Update’ method in thescript attached to the virtual camera object might rotate the camera'stransform by 1 degree around the y axis in the scene coordinatereference frame. In this way, the virtual camera would rotate slightlywith each frame update, producing the sensation of motion.

The Unity graphics engine also allows scripts to access various sensorsand input devices. For example, the Unity graphics engine provides anApplication Programming Interface (API) for accessing user controls suchas a keyboard and determining if particular keys of a keyboard or areasupon a touch sensitive display screen are pressed. In a similar manner,mouse movements and clicks, and joystick movements and button pressescan be accessed. Also, the Unity graphics engine provides a method forobtaining readings from sensors (e.g., gyroscope sensors andaccelerometer sensors) of a user device 10. Alternatively, readings fromthe sensors may be obtained using code of the operating system of theuser device 10. Data from different ones or combinations of these inputsmay be used in different embodiments to provide logic in scripts tochange the way the virtual scene appears in the different views.

Continuing the example above, a script could be written with an ‘Update’method that queries the keyboard to determine if the left-arrow′ key isdepressed. If the key is depressed, the script could update the camera'stransform to rotate it one degree about the y axis. This provides ameans for the user to control the orientation of the virtual camera inthe virtual scene and generation of different views by pressing theleft-arrow′ key.

Traditionally, users have interacted with 3D graphics applications usinga mouse and keyboard. Many modern user devices include an applicationprogramming interface (API) for accessing sensor data from the sensorsof the user device 10. As discussed in example embodiments below, thissensor data can be used to interact with the 3D environment of thevirtual scene displayed in different views on the display screen of theuser device 10. Some example embodiments describe apparatus and methodswhich allow the user to feel more immersed in the virtual environmentcompared with other interaction methods while enabling the user to besimultaneously connected to and taking cues from the physicalenvironment.

In one embodiment, processing circuitry 34 controls the generation ofthe plurality of different views of the virtual model as the usernavigates the virtual model. The views are drawn in a plurality ofrespective frames and may show different portions of the virtual modelas the user navigates the virtual model. As described below, user inputsentered via user controls of the user device including the graphicaluser interface (or other appropriate interface such as a mouse orkeyboard of the user device) and/or inputs generated by physicalmovements (e.g., rotations and translations) of the user device areutilized to navigate the virtual model and control the differentportions of the virtual model which are presented in the differentviews. The redrawing of frames upon the display device 32 issufficiently fast in at least one embodiment to create a video playbackexperience to the user while navigating the virtual model. In someembodiments, the views of the virtual model are adjusted in real time asthe user enters inputs into the user device via user controls and/ortranslates or rotates the user device.

Referring to FIG. 1A, another virtual scene 12 a is shown resulting fromthe virtual camera being positioned at a different view point and viewdirection compared with the view point and view direction of the virtualcamera used to draw the frame of FIG. 1. In the illustrated example ofFIG. 1A, the view point of the virtual camera has been moved closer tothe furniture object and the view direction of the virtual camera hasbeen rotated to the left compared with the virtual scene 12 of FIG. 1.

Referring to FIG. 2, one embodiment of hardware components of acomputing system or device which may be utilized as a user device 10 isshown. In the illustrated example embodiment, the user device 10includes a display screen 32, processing circuitry 34, storage circuitry36, a camera 38, location sensors 40, gyroscope sensors 42,accelerometer sensors 44, and communications circuitry 46. Otherembodiments of the computing system or device are possible includingmore, less and/or alternative components. For example, the user devicemay include a magnetometer to measure the alignment of the device withrespect to the Earth's magnetic field and a range camera which providesdistance information for pixels of captured images.

Display screen 32 is configured to interact with a user includingconveying data to a user (e.g., depicting visual images for observationby the user) as well as receiving inputs from the user, for example, viaa touch sensitive screen interface. Display screen 32 is configured as auser input device or control in the form of a graphical user interface(GUI) and which may directly receive user entered inputs in oneembodiment. Although not shown, the user device may include additionalassociated user input devices or controls, such as a mouse, keyboard,etc., for directly receiving user entered inputs as mentioned above.

In one embodiment, processing circuitry 34 is arranged to process data,control data access and storage, issue commands, control the display ofimages. Processing circuitry 34 may additionally control otheroperations discussed herein including navigation of virtual models anddisplaying different views of the virtual models during navigation.

Processing circuitry 34 may comprise circuitry configured to implementdesired programming provided by appropriate computer-readable storagemedia in at least one embodiment. For example, the processing circuitry34 may be implemented as one or more processor(s) and/or other structureconfigured to execute executable instructions including, for example,software and/or firmware instructions. Other example embodiments ofprocessing circuitry 34 include hardware logic, PGA, FPGA, ASIC, statemachines, and/or other structures alone or in combination with one ormore processor(s). These examples of processing circuitry 34 are forillustration and other configurations are possible.

In some embodiments described herein, processing circuitry 34 isconfigured to detect movements (e.g., rotational and/or translationalmovements) of the user device 10 and to generate different views showingdifferent virtual content of the virtual model as a result of thedetected movement and which may correspond to the detected movement. Insome embodiments, movement of the user device may be detected bymonitoring outputs from accelerometer, gyroscope, and location sensors.In addition, a change in orientation or rotation may be detected orcontrolled along one or more of the axes of the respective coordinatereference frame.

Storage circuitry 36 is configured to store programming such asexecutable code or instructions (e.g., software and/or firmware),electronic data, model data of virtual models (e.g., architecturalstructures) to be navigated and interacted with and which is used tocreate different views of the virtual models during user navigation,databases, image data from camera 38 (or from other sources of contentof the physical world, etc.) or other digital information and mayinclude computer-readable storage media.

At least some embodiments or aspects described herein may be implementedusing programming stored within one or more computer-readable storagemedium of storage circuitry 36 and configured to control appropriateprocessing circuitry 34.

The computer-readable storage medium may be embodied in one or morearticles of manufacture which can contain, store, or maintainprogramming, data and/or digital information for use by or in connectionwith an instruction execution system including processing circuitry 34in one embodiment. For example, computer-readable storage media may benon-transitory and include any one of physical media such as electronic,magnetic, optical, electromagnetic, infrared or semiconductor media.Some more specific examples of computer-readable storage media include,but are not limited to, a portable magnetic computer diskette, such as afloppy diskette, a zip disk, a hard drive, random access memory, readonly memory, flash memory, cache memory, and/or other configurationscapable of storing programming, data, or other digital information.

Camera 38 is configured to generate image data of the physical worldcontent including still photographs and video data and which may bestored using storage circuitry 36. In some embodiments described below,processing circuitry 34 may insert image data generated by the camerainto views of the virtual model which are displayed using the displayscreen 32.

Location sensors 40 use GPS or triangulation of data network nodes inexample embodiments to determine the user device's global location inthe physical world, i.e., the global coordinate reference frame.

Gyroscope sensors 42 measure the orientation of the user device withrespect to the global coordinate reference frame in the physical worldas well as changes in the orientation of the user device. In oneembodiment, three gyroscope sensors 42 monitor rotations about threeorthogonal axes and provide gyroscope data indicative of the rotationsof the user device 10 about these axes.

In one embodiment, three accelerometer sensors 44 sense accelerationsalong the three coordinate axes of the device coordinate reference frameof the user device.

Communications circuitry 46 is arranged to implement communications ofuser device 10 with respect to external devices (not shown). Forexample, communications circuitry 46 may communicate informationbi-directionally with respect to user device 10 and include hardware forwired connections (e.g., network interface card (NIC), serial orparallel connections, USB port, Firewire interface), and/or circuitryfor wireless connections (e.g., Bluetooth, Cellular, GPS, WiFi, etc.).

As described below in some embodiments, the user device uses data inputsfrom these example sensors to allow a user to view and navigate avirtual model in a more natural way than can be achieved through simplekeyboard, mouse or touch screen manipulation.

Referring to FIG. 3, an example camera 50 (such as camera 38, or avirtual camera utilized to generate views of a virtual model) is shown.A camera's field of view (FOV) defines how much of the scene (real orvirtual) is visible in an image captured by the camera. Field of view(FOV) of the camera 50 is a square pyramid 52 in one example with itsapex at the camera 50 and its axis parallel to the view direction of thecamera. The shape of the pyramid is defined by two parameters includingan angle (theta) between opposing triangular faces (usually the top andbottom faces) and an aspect ratio for the resulting image. The aspectratio constrains the angle between the other two faces. In order toavoid distortion in the displayed image, the aspect ratio is generallychosen to match the aspect ratio of the display screen used to displaythe image.

As mentioned above, a virtual camera is used to control the generationof views of the virtual model in one embodiment. Various parameters ofthe virtual camera define how the views are generated. For example, theview point of the virtual camera specifies where the virtual camera ispositioned within a virtual model relative to other objects of thevirtual model. A view direction 53 specifies the direction in which thevirtual camera is pointing in the virtual scene. Additional informationmay be provided regarding the orientation or rotation of the virtualcamera within the virtual scene (e.g., information regarding azimuthrotations, elevation rotations, and twist rotations). The view direction53 of the virtual camera is a vector emitted from the view point of thevirtual camera at an orientation which is as specified by theorientation of the virtual camera in one embodiment.

Referring to camera 38 of the user device, the view direction of thecamera 38 of the user device is a vector which is normal to the lens andpasses through the center of the lens in one embodiment.

Different methods for interacting with the virtual model are provided indifferent embodiments. User inputs via user controls of the user deviceand/or data from sensors and circuitry of the user device controlnavigation of a virtual model as described in the example embodimentsbelow. For example, these inputs change one or more of the view point(location of the virtual camera within a virtual scene) and view angle(orientation of the virtual camera within the virtual scene). Thesechanges result in different views of the virtual model being displayedvia the display screen 32.

Different operational modes utilize different ones of the inputs tocontrol navigation of the virtual model including changing the viewpoint and view direction of the virtual camera within the virtual sceneof the virtual model.

Referring to FIG. 4, a view 60 of a virtual model which may be displayedusing the display screen of the user device is shown. The image of theview 60 generated upon the display screen also includes user controls inthe form of joysticks 62, 63 in one operational mode. Gestures of auser's finger may interact with the joysticks 62, 63 upon the screendisplay to navigate the virtual model including changing the views ofthe virtual model. In one example, joystick 62 controls translationmovements of the virtual camera in two degrees of freedom within thevirtual scene and joystick 63 controls rotation movements of the virtualcamera in two degrees of freedom within the virtual scene. In oneimplementation, the change in the position of the virtual camera isproportional to the displacement of the joystick from its home position.In other words, pushing the joystick further forward produces asensation of translating or rotating faster.

In other operational modes, translation and rotational movements of theuser device are detected and used to control navigation of the virtualmodel (alone or in combination with inputs entered by the user usingcontrols).

Referring to FIG. 5, a user is shown holding a user device 10, such as atablet computer. During typical operation, the user may hold the userdevice 10 in front while observing images generated by the displayscreen 32 which include different views of the virtual model as the usertranslates and/or rotates the user device.

The user device 10 defines a view direction 72 which is normal to aplane which includes the display screen 32 and emanates outwardly fromthe user device in a direction opposite to the direction in which imagesdisplayed using the display screen 32 are viewed by the user in oneembodiment. Movements of the user device 10 by the user in the physicalworld may be detected by one or more of the location sensors 40,gyroscope sensors 42, and accelerometer sensors 44 and used to navigatethe virtual model including control of translation and rotation of thevirtual camera within the virtual scene.

The rotational movements of the user device 10 change the view direction72 of the device 10. For example, in FIG. 5, the user has rotated theuser device 10 to the left and increased the elevation angle resultingin a change of the view direction 72 a to the left and upwards. The viewdirection 72 of the user device 10 controls the view direction of thevirtual camera within the virtual scene in some embodiments as describedin further detail below. For example, data regarding the orientation orchanges in orientation of the user device 10 (and view direction 72) inthe physical world may be used to provide a corresponding orientation orchanges in orientation of the virtual camera in the virtual scene. In anillustrative example, a user rotating with the user device 10 to theleft 40 degrees in the physical world is detected by the gyroscopesensors 42 and used to control leftward rotation of the virtual camera40 degrees within the virtual model.

In one embodiment, the orientation of the virtual camera is controlledor defined to be at an orientation within the scene coordinate referenceframe of the virtual model which is the same as the orientation of theuser device in the physical coordinate reference frame of the physicalworld (the view direction of the user device is the same as the viewdirection of the user device).

In one example embodiment, rotational offsets of the user device withrespect to an initial orientation are calculated based upon subsequentrotations of the user device 10 and are used to control correspondingrotations of the virtual camera from an initial orientation to anupdated orientation corresponding to the updated orientation of the userdevice.

In addition, the view direction 72 of the user device 10 may alsocontrol a direction of translation of the virtual camera within thevirtual scene in some embodiments. In one example of this embodiment,changes in translations according to the view direction 72 of the userdevice 10 result in elevational changes of the virtual camera within thevirtual scene (downward if the user device 10 is pointed belowhorizontal or upward if the user device 10 is pointed above horizontal).

In one example of this mode where the view direction of the virtualcamera is controlled using rotations of the user device 10, forward andbackward translations of the virtual camera in the view direction of thevirtual camera are controlled by user inputs to the user controls (e.g.,forward and backward movements of the translation joystick 62). As theuser moves his finger to move the virtual joystick 62 forward, thevirtual camera is translated forward along the view direction of thevirtual camera within the virtual scene in the scene coordinatereference frame. If the view direction of the user device is pointingupwards above horizontal in the global coordinate reference frame, thevirtual camera is translated upwards towards the ceiling of the virtualmodel in one illustrative example.

Accordingly, the elevation of the virtual camera within the virtualmodel may change during this operational mode. If the user points theuser device as shown in FIG. 5 (downward relative to horizontal), andalso moves joystick 62 forward, the virtual camera is translated forwardand downward within the virtual scene corresponding to the downwardangle of the view direction 72 and which would result in the virtualcamera being positioned closer to the floor of the architecturalstructure (i.e., a lower elevation). Likewise, the user device 10 may bepointed upward so the view direction 72 is in a direction abovehorizontal and the elevation of the virtual camera is increased withinthe virtual model if the joystick 62 is also moved forward.

In this example operational mode, the view direction and translationdirections of the virtual camera within the virtual scene are equivalentand the user changes viewing orientation of the virtual camera byrotating the user device while translation is controlled by displacingjoystick 62 as described above.

Moving the virtual camera through the virtual scene according to thisexample operational mode produces the sensation of ‘flying’ through thescene. When this mode is selected and entered by the user, the virtualcamera is placed at some location (view point) and orientation relativeto the virtual scene. The gyroscope sensors 42 detect changes inorientation of the user device and view direction 72 and these changesare used to modify the orientation (and view direction) of the virtualcamera relative to the virtual scene. The virtual camera is updated insuch a way that the user perceives the mobile device as a window lookinginto the virtual scene. As the user changes the orientation of the userdevice, the user device displays the appropriate portion of the virtualscene (e.g., if the device is held in front of the user and rotated tothe left in the physical world, the virtual camera rotates to the leftin the virtual model).

As the user displaces the joystick 62 forward and changes theorientation of the device 10 to the left (and the view direction 72),the virtual camera translates in an arc to the left within the virtualscene. ‘Looking up’ with the user device 10 and pressing the virtualjoystick 62 forward ‘moves’ the user toward the ceiling or sky. ‘Lookingdown’ and displacing the joystick 62 backward produces the sensation ofzooming out on the virtual scene from above. The display screen 32behaves as though it were a transparent piece of glass through which thevirtual scene of the virtual model is viewed in this describedembodiment.

In another operational mode, the rotations of the virtual camera arecontrolled by rotations of the user device 10 as described above, andtranslations of the virtual camera are controlled to be at a constantelevation within the scene coordinate reference frame. In this example,the azimuth direction 74 of the user device 10 in the global coordinatereference frame controls the azimuth direction of translations of thevirtual camera in the virtual scene at a constant elevation. Thetranslation direction 74 may be considered to be a projection of theviewing direction 72 and may be used to define the azimuth direction oftranslations within a translation plane which is parallel to horizontalwithin the global coordinate reference frame at the desired constantelevation.

The joystick 62 may be used to control the translations of the virtualcamera in the virtual scene including forward and backward translations(responsive to respective forward and backward inputs via the joystick62) along the view direction of the virtual camera at the constantelevation within the virtual scene in this described embodiment.

As mentioned above for the presently described operational mode, thevirtual camera is translated at an elevation of the virtual camera whichis fixed relative to the base plane (e.g., floor) of the virtual model.In one embodiment, the virtual camera is positioned at 56″ relative tothe base plane to resemble the eye level of a human. Accordingly, inthis example embodiment, virtual camera translations are intended tomaintain the elevation of the virtual camera constant within the virtualscene even if the user device 10 is pointed up or down relative tohorizontal while moving.

This embodiment more closely aligns with the experience of walking wherethe user's view point remains at a constant elevation above the terrain.The elevation of the virtual camera changes only if the elevation of theunderlying scene changes in this example implementation. For example, ifthe user reaches a staircase in a virtual model of a home, the elevationof the virtual camera changes to represent ascending or descending thestaircase.

There are a number of ways to provide the virtual camera at the properelevation within the virtual model. In one embodiment, a movement vectorwhich defines the movement of the virtual camera is calculated using thecurrent view direction 72 of the user device and the vector is projectedonto a plane whose normal points in the global Z direction (assuming theglobal Z coordinate represents elevation in the scene) and is at thedesired elevation.

Another method is to translate the virtual camera to the new view pointand examine the height of the camera above the terrain in its newposition. If the new elevation is not the expected constant value, asecond translation in the global Z direction is performed to move thevirtual camera to its proper height.

In addition, a look-ahead point (e.g., a point offset some distance fromthe virtual camera position in the view direction) is maintained and anelevation calculation to position the virtual camera can be performedfor the look-ahead point and the virtual camera's change in elevation isinterpolated between the elevation at the current location and thelook-ahead point in another example. This embodiments may be used toreduce the presence of unnatural jumps in the view point when theelevation changes abruptly (e.g., at a staircase within the virtualmodel).

Another example method may utilize metadata embedded in the virtualmodel. This metadata exists at elevation transition points (e.g., at thetop and bottom of a staircase) in some embodiments. When the virtualcamera reaches an area of the scene containing metadata, the metadata isused to define how the elevation changes to the next transition point.

In addition, the sensors of the user device 10 may be used to determinethe elevation of the virtual camera in the scene in another embodiment.For example, data from a range camera of the user device and thegyroscope sensors 42 are combined to find the length and angle of therange camera's direction vector and this vector forms the hypotenuse ofa right triangle, with the elevation as one of its legs. The elevationmay be calculated using straightforward trigonometry if the length andangle of the hypotenuse are known. In one embodiment, a suitable rangecamera is configured to provide depth or distance information ofindividual pixels of images to the lens.

In addition to use of natural rotations of the user device 10 to controlthe view direction of the virtual camera, translations of the virtualcamera within the virtual model may also be controlled using informationregarding natural translations of the user device in the physical worldin one embodiment. Also, translations of the virtual camera in thevirtual scene may be controlled using natural translations of the userdevice in the physical world in combination with use of the usercontrols (e.g., joystick 63) to control orientation in some embodiments.Furthermore, in one arrangement, the translation movements of thevirtual camera in the virtual scene correspond to translation movementsof a user with the user device in the physical world.

In a more specific example, a distance the virtual camera is translatedin the virtual scene corresponds to a distance the user device istranslated in the physical world. In one example, the scales of thevirtual model and the physical world are the same (e.g., a translationof the user device a distance of one foot in the physical world resultsin a translation of the virtual camera a distance of one foot in thevirtual model). In other embodiments, the virtual model and physicalworld may be scaled differently (e.g., the virtual camera is moved oneyard in the virtual model if the user device is moved one foot in thephysical world).

For example, the user might physically walk through an open field orgymnasium and see the virtual scene on the screen update as they do so.This mode might be used for a walkthrough of a new house plan orhistoric structure in more specific examples. The scale may corresponddistances within the virtual model to be the same as the physical worldor the scale may be changed to allow the user to walk through virtualmodels of something very small (the human circulatory system) or verylarge (the Grand Canyon). Regardless of the scale of the scene,correlating physical movements (e.g., rotations and/or translations)with respective virtual movements provides a sense of proportion that isdifficult to achieve by moving a joystick or mouse or pressing keyboardbuttons.

In some examples described below, the orientation of the virtual camerais controlled by orientation data received from the gyroscope sensors 42(natural rotation of the user device). In other embodiments, theorientation of the virtual camera is controlled using joystick 63 orother user control for receiving user inputs.

In one embodiment, the sensors of the user device 10 can be used tonavigate the virtual scene as the user changes his physical positionwith the user device 10 (e.g., while the user holds, wears, or otherwiseconnected with the user device 10). For example, translations of theuser device 10 may be monitored and detected via inputs from the sensorsthat provide information regarding the location of the user device 10(and user) in space (e.g., location sensors 40, gyroscope sensors 42,accelerometer sensors 44, range camera, etc.).

In particular, data generated and output by the sensors controlmovements of the virtual camera and are used to update the position ofthe virtual camera in the virtual scene in this example embodiment.Detected movement of the user device in the physical world a givendistance from a first location to a second location with the globalcoordinate reference frame may control corresponding movement of thevirtual camera the same (or scaled) distance of the virtual camera inthe scene coordinate reference frame of the virtual scene. Thecorresponding translated movement of the virtual camera within thevirtual scene changes or updates the view point of the virtual camerawithin the virtual scene.

In some embodiments described below (e.g., the method of FIG. 7regarding use of natural translation of the user device), a distance theuser device moves in the physical world is determined by on boardsensors (gyroscope sensors 42 and/or accelerometer sensors 44) withoutuse of any signals which were originally transmitted externally of theuser device and received by the user device (e.g., GPS) and withoutprocessing of images captured using the camera.

In one illustrative example, gyroscope data from gyroscope sensors 42and acceleration data from accelerometer sensors 44 are used to detect auser moving with the user device 10, such as stepping. In oneembodiment, footsteps of the user holding the user device 10 aredetected and used to control translation of the virtual camera. Anexample method of detecting a step of a user holding the user device 10and which detection results in translation of the virtual camera withinthe virtual model is described below with respect to FIG. 7. In someembodiments, the view direction of the user device in the physical worldcontrols the view direction of the virtual camera in the virtual scene.For example, if a wall object of the virtual model is shown using thedisplay screen, the detection of a step results in the change of theview point of the virtual camera in a forward direction toward the wallobject which is displayed and the wall object is closer in the updatedview in an updated frame following the detection of the footstep. In oneembodiment, the elevation of translated movement of the virtual camerain accordance with the view direction of the user device is controlledto be constant. This embodiment provides the user with the impression oflooking through a piece of glass at the virtual model as all scenenavigation is controlled through physical movement.

Accordingly, as described above in some illustrative embodiments,movements of the virtual camera within the virtual scene correspond tomovements of the user device in the physical world, or put another way,the translation and/or rotational movements of the user device in thephysical world control corresponding translations and rotations of thevirtual camera in the virtual scene.

As discussed above, some operational modes combine aspects of the user'sphysical environment with the virtual environment and some embodimentsenable the initial orientation and view point of the virtual camerawithin a virtual scene to be specified by the user as navigation of thevirtual model is initiated.

In one embodiment, a user input may specify the initial orientation andview point of the virtual camera. For example, the user taps a locationin a 2D plan view of the virtual scene to define the view point anddrags their finger along the display screen 32 to define the viewdirection in the virtual scene. The initial 3D view of the virtual sceneis drawn with the virtual camera positioned at the tapped location atsome default elevation and pointing in the dragged direction in oneembodiment. In one example, the user may set the initial view point andinitial orientation of the virtual camera looking at the front door, andthereafter, rotations of the user device to the right would result inthe front door moving to the left in the virtual scene.

Another method allows the user to enter a 3D view of the virtual sceneand use the joysticks 62, 63 to adjust the view point and orientation ofthe virtual camera to suit. Thereafter, the joysticks 62, 63 may remainor be removed from the view and natural orientation and/or translationvia the user device may subsequently control the view point andorientation of the virtual camera following the initial placement.

Another method allows the user to specify a latitude, longitude, andorientation direction for the virtual model. When entering a modecontrolled by natural motion (either rotations or translations with theuser device), the user device may determine the latitude, longitude, andorientation of the user device in the global coordinate reference frameand compute offsets from the specified values with respect to the scenecoordinate reference frame and which may be used to place the virtualcamera in its initial location and orientation in the virtual scene.

The discussion continues below with respect to example implementationsfor controlling rotations and translations of the virtual camera withinthe virtual scene using natural rotational and translation movements ofthe user device in the physical world.

In one embodiment of the user device, three gyroscope sensors areconfigured to monitor rotations about three orthogonal axes with respectto a global coordinate reference frame and three accelerometer sensorsmonitor accelerations along the three orthogonal axes of the devicecoordinate reference frame. Data generated by the sensors regarding userdevice orientation and translation may be utilized to navigate andobserve different views of the virtual model as mentioned above.

Accordingly, one or more movements of the user device (e.g., changingthe orientation of the user device and/or translating the user device)may be used to interact with and observe different views of the virtualmodel in one embodiment. In a more specific example mentioned above,changes in orientation and translations of the virtual camera which isused to view the virtual model may be controlled corresponding tochanges of orientation and translations of the user device (e.g., thevirtual camera is rotated to the left in the scene coordinate referenceframe as a result of the user device being rotated to the left in theglobal coordinate reference frame).

The gyroscope sensors 42 allow the user device to report how it isoriented with respect to a global coordinate reference frame defined bythe Earth's gravity and magnetic fields. When an application running onthe user device requests the ability to obtain orientation data, theuser device begins sampling the gyroscope sensors at regular intervals(60 times per second, for example) in one embodiment. At each sample,the gyroscope sensor provides gyroscope data by notifying a clientapplication being executed by the user device of the current orientationof the user device with respect to the global coordinate reference framein one implementation. The gyroscope data of these notifications come inthe form of 3 Euler angles, a quaternion, or a transformation matrix insome examples. Regardless of its form, the notification containsinformation about the offset of the three coordinate directions of thepresent orientation in the device coordinate reference frame from thethree coordinate directions of the global coordinate reference frame.

In one embodiment, an initial orientation of the user device isdetermined in the global coordinate reference frame. Thereafter, theuser may change the orientation of the user device, and the user devicecalculates offset rotations of the user device from the initialorientation of the user device as the user rotates the device, and theseoffset rotations may be used to update the orientation of the virtualcamera within the virtual scene in one example (i.e., rotate the virtualcamera a certain number of degrees about an axis of the scene coordinatereference frame the same number of degrees of the offset which the userdevice rotated about a corresponding axis in the global coordinatereference frame). In one embodiment, the processing circuitry isconfigured to calculate a plurality of offsets of the view direction ofthe user device compared with an initial view direction of the userdevice, and to use the offsets to control the view direction of thevirtual camera at the first and second orientations.

Referring to FIG. 6, a flow chart shows a method according to oneembodiment for updating orientation of the virtual camera in the virtualscene based on the processing of gyroscope data generated by the userdevice. The processing of the gyroscope data may be utilized to changerotations of the virtual camera within the virtual scene as a result ofcorresponding rotations of the user device in the physical world. Themethod may be executed by processing circuitry of the user deviceaccording to one embodiment. Other methods are possible including more,less and/or alternative acts.

In this described example, the gyroscope data provided by the gyroscopesensors can be used to control an orientation of the transform of thevirtual camera object within the virtual scene and which is used tochange the view of the virtual model.

In one embodiment described below, the processing circuitry uses achange in orientation of the user device in the global coordinatereference frame to determine an offset within the scene coordinatereference frame and to use the offset to provide the virtual camera atan updated orientation from a previous orientation and which correspondsto the change in orientation of the user device. In one embodimentdescribed below, the offset provides a rotation of the virtual camera inthe scene coordinate reference frame which is the same as a rotation ofthe user device in the global coordinate reference frame (e.g. leftwardazimuth rotation of the user device 10 degrees in the physical worldprovides leftward azimuth rotation of the virtual camera 10 degrees inthe virtual world). In one embodiment, the azimuth, elevation and twistof the virtual camera within the virtual scene are controlled in thefirst and second views using the azimuth, elevation and twist of theuser device in the physical world

When the user enters an interactive mode that uses the gyroscope sensorsto control orientation, the user device's current (instantaneous)orientation with respect to the global coordinate reference frameR_(di), and the camera object's current orientation with respect to thescene coordinate reference frame, R_(di) are recorded at an act A2.These values are used to compute and cache initial rotations of the userdevice with respect to the global coordinate reference frame (R_(d0))and initial rotations of the virtual camera object (R_(c0)) with respectto the scene coordinate reference frame as described with respect toacts A6-A11.

At an act A4, it is determined whether rotations of the user device(R_(d0)) and the camera object (R_(c0)) have been initialized (i.e.,found).

If the condition of act A4 is affirmative, the process proceeds to anact A13.

If the condition of act A4 is negative, the process proceeds to performinitialization of rotations of the user device (R_(d0)) and the cameraobject (R_(c0)) at acts A6-A11.

At an act A6, elev_(d) and elev_(c) are determined where elev_(d) is theangle between the z axis of the global coordinate reference frame(global Z axis) and the view direction of the user device, and elev_(c)is the angle between the z axis of the scene coordinate reference frameand the view direction of the virtual camera.

At an act A8, the virtual camera's rotation about the x axis of scenecoordinate reference frame 19 of FIG. 1 (R_(elevation)) can be foundusing equation 1:R _(elevation) =E(elev_(d)−elev_(c),0,0)  (1)

In equation 1, E(elev_(d)−elev_(c), 0,0) is a rotation defined by Eulerangles. An Euler angle rotation is achieved by specifying rotationangles about the three orthogonal coordinate directions of a referenceframe. The three rotations are applied in a specific order to produce aresultant overall rotation. The order in which the rotations are appliedis important, since changing the order of these rotations produces adifferent resultant rotation. Unity implements an Euler angle rotation,E(α, β, γ), by first rotating the object by γ degrees about the Z axis,then by a degrees about the X axis, and finally by β degrees about the Yaxis. In Unity, it is possible to specify that the rotations take placeabout the scene coordinate axes or about the object's local coordinateaxes. In either case, the rotations are performed about fixed axes,rather than about axes that may have changed as a result of one or moreof the previous Euler component rotations. For example, rotating anobject about its local Z axis by γ degrees causes the object's X and Yaxes to point in new directions. Unity applies the next rotation, α,about the object's original X axis rather than the new X axis.

At an act A9, the twist angle of the user device (twist_(d)) is found bydetermining the angle between the global Z axis and a vector pointingout of the right side of the device (i.e., x axis of the devicecoordinate reference frame 21 of FIG. 1) and subtracting this angle from90 degrees.

At an act A10, the user device's twist orientation (R_(twist)) iscomputed. R_(twist) can be found using equation 2:R _(twist) =E(0, 0,twist_(d))  (2)

At an Act A11, the initial rotations of the user device (R_(d0)) and thecamera object (R_(c0)) are computed and cached. R_(d0) is simply set toR_(di), as in equation 3.R_(d0)=R_(di)  (3)

Three components of the orientation of the virtual camera are consideredto calculate R_(c0) including the virtual camera's azimuth angleR_(azimuth), its elevation angle R_(elevation), and its twist angleR_(twist). For a desirable user experience, the virtual camera objectmaintains its current azimuth within the virtual scene when agyroscope-controlled interactive mode is entered in one embodiment.

However, the elevation and twist of the virtual camera are controlled tohave the same elevation and twist in the scene coordinate referenceframe as the elevation and twist of the user device in the globalcoordinate reference frame when entering gyroscope-controlledinteractive mode in one embodiment. This is accomplished with equation 4in the described embodiment.R _(c0) =R _(ci) ·R _(elevation) ·R _(twist)  (4)

Following the completion of the initialization of acts A6-A11, themethod proceeds to an act A13 to perform updates to the virtual cameratransform which may change the orientation of the virtual cameraresulting from rotations of the user device and are used to drawrespective frames including different views of the virtual model atdifferent moments in time, such as a sequential video.

At an act A13, the device offset rotation (R_(dΔ)) is computed from thedevice's original and current orientations during frame updates bycombining the inverse of the initial device rotation R_(d0) with thecurrent device rotation R_(di) in equation 5.R _(dΔ) =R _(d0) ⁻¹ ·R _(di)  (5)

At an act A14, a new camera object rotation (R_(cnew)) is computedduring frame updates by applying the device offset rotation R_(dΔ) tothe initial rotation of the virtual camera R_(c0) in equation 6.R _(cnew) =R _(dΔ) ·R _(c0)  (6)

The device offset rotation R_(dΔ) indicates updates or changes to bemade to the orientation of the virtual camera in the scene coordinatereference frame corresponding to the changes of the orientation of theuser device in the global coordinate reference frame.

At an act A15, the graphics engine sets the orientation of the cameraobject's transform to R_(cnew) (which changes as the user deviceorientation changes) during each frame update and which results inchanges in orientation of the view direction of the virtual camerawithin the virtual scene and corresponding to changes of the orientationof the user device in the physical world.

Updating the camera object in this way produces the sensation of lookingthrough the device into the virtual scene. As the user rotates the userdevice, the view direction of the virtual camera updates allowing theuser to easily explore the scene. The rotation of the device to the leftwill result in the virtual scene moving to the left (e.g., for a virtualscene within a room, rotation of the user device to the left may show awall of the room to the left which was not shown in the previous viewsof the virtual scene).

At an act A16, the frames of the virtual scene are drawn using the userdevice.

The gyroscope data from the gyroscope sensors 42 may also be used totransform accelerometer data from the device coordinate reference frameto a global coordinate reference frame in one embodiment.

Accelerometer data can be obtained from the accelerometer sensors 44 ofthe user device at the same time that orientation notifications occurwhich include the gyroscope data from the gyroscope sensors 42 in oneembodiment.

The accelerometer data is provided in device coordinates and the userdevice registers accelerations along the three axes of the user deviceregardless of how the device is oriented in space. However, it may bemore useful for some applications to utilize acceleration informationwith respect to the global coordinate reference frame of the physicalworld. In one embodiment, a transformation matrix is provided within anorientation update sample from the gyroscope sensors and which can beused for this purpose.

Mathematically, acceleration a may be represented as a 3-dimensionalvector with one element for each of the coordinate axes of therespective reference frame per equation 7.

$\begin{matrix}{a = \begin{pmatrix}a_{x} \\a_{y} \\a_{z}\end{pmatrix}} & (7)\end{matrix}$

Global acceleration with the global coordinate reference frame,a_(Global), can be calculated from the acceleration of the user devicein the device coordinate reference frame, a_(Device), and thedevice-to-global transformation matrix T_(DeviceToGlobal) using equation8.a _(Global) =T _(DeviceToGlobal) ·a _(Device)  (8)

During updates, the accelerometer sensors of the user device generatea_(Device) and the gyroscope sensors generate the global-to-devicetransformation matrix, T_(GlobalToDevice). The device-to-globaltransformation matrix T_(DeviceToGlobal) can be obtained from theinverse of the global-to-device transformation matrix T_(GlobalToDevice)using equation 9.T_(DeviceToGlobal)=T_(GlobalToDevice) ⁻¹  (9)

In general, computing the inverse of a matrix is problematic since thisoperation can be computationally intensive and because the matrix mightbe singular. However, in this described embodiment, the matrix is a purerotation matrix since it only contains input from the device's gyroscopesensors and a pure rotation matrix is not singular, and its inverse isits transpose. Accordingly, the device-to-global transformation matrixT_(DeviceToGlobal) may be calculated (transposed) using equation 10 inone example.T_(GlobalToDevice) ⁻¹=T_(GlobalToDevice) ^(T)  (10)

Thus, the global acceleration a_(Global) can be obtained by Equation 11.a_(Global)=T_(GlobalToDevice) ^(T)·a_(Device)  (11)

This example method is efficient since computing the transpose of amatrix is less computationally intensive than computing its inverse andthe computation is performed for the samples being generated many timesper second (e.g., 60 Hz) in one embodiment.

As discussed above, orientation of the user device in the physical worldmay be used to control the orientation of the virtual camera withrespect to the virtual model in one embodiment. In addition, locationand movement of the user device in the physical world may also be usedto control the location and movement of the virtual device with respectto the virtual model in one embodiment.

In one embodiment, one or more location sensor of the user device mayprovide location data which may be utilized to determine a device'slocation and velocity in space. The processing circuitry combines datafrom the location sensors (e.g., GPS and/or data network signals) toproduce an estimate of the device's location in one embodiment. Readingsfrom location sensors are desirable because they are given as absolutecoordinates in the global coordinate reference frame rather than asoffsets from previous readings. Use of location sensors isself-correcting because slight errors in the sensor readings for onesample do not affect subsequent samples. The location data obtained bythe location sensor may be used to control translation of the virtualcamera within the virtual model according to one embodiment.

In example embodiments, the virtual model is at least partially orcompletely aligned with the physical world. If completely aligned, arelative offset between the scene and global coordinate reference scenesis reduced or eliminated such that locations in the virtual model may bemapped to corresponding locations within the physical world and thevirtual camera is provided at a location within the virtual model whichcorresponds to a location of the user device in the physical world. Inaddition, the virtual camera may be moved within the virtual modelcorresponding to movements of the user device in the physical world asdetected by the location sensors. Accordingly, location data from thelocation sensors may be used to control the location and movements ortranslations of the virtual camera within the virtual model in anexample embodiment. In one embodiment, the location of the user devicemay be determined at individual frame updates and detected movement ofthe user device may be used to move the virtual camera to a new viewpoint (as well as make adjustments to the orientation of the virtualcamera corresponding to changes in orientation of the user devicebetween frames).

In some embodiments, coordinates of the virtual scene may be alignedwith physical coordinates of the physical world as mentioned above. Forexample, an xy base plane of the virtual scene is the same as the xyplane of the physical world on the ground in some embodiments (e.g., thefloor of an architectural structure virtual model is on the ground). Insome embodiments, the virtual scene and physical world may be completelyaligned (i.e., a North direction in the virtual model is aligned withNorth in the physical world). In one example embodiment, a user may holdthe device facing a given direction in the real world and use joystick63 to rotate the virtual scene until the desired portion of the virtualmodel is displayed in the view to align the virtual and globalcoordinate reference frames. The orientation of the virtual camera isthe same as the orientation in the physical world in this example.

As discussed in some example embodiments herein, the scene coordinatereference frame may be partially or completely aligned with the globalcoordinate reference frame. Furthermore, as discussed above, the userdevice and virtual camera are initially coupled with one another bystoring initial orientations or rotations of the user device and virtualcamera, and thereafter offsets are generated resulting from movements ofthe user device during navigation and which are applied to the rotationof the virtual camera to synchronize movements of the user device andthe virtual camera with one another in the global and scene coordinatereference frames which may be partially or completely aligned with oneanother. Using the above described example method in some embodiments,the orientations of the virtual camera within the scene coordinatereference frame correspond to or are the same as the respectiveorientations of the user device in the global coordinate referenceframe.

At any time, the user can adjust the alignment of the scene and globalcoordinate reference frames to change their relative offset with respectto one another (e.g., using a joystick to rotate and change the azimuthof the virtual model with respect to the physical world). Thereafter, anoffset rotation of the user device from an initial orientation isdetermined as a result of changes in orientation of the user device foreach frame update, and the same offset rotation is applied to change theorientation of the virtual camera with respect to its initialorientation. As discussed herein, the orientation of the user device maybe monitored using the view direction of the user device and theorientation of the virtual camera may be controlled with respect to theview direction of the virtual camera in one embodiment.

Accordingly, in one embodiment, a user may orient or align the virtualscene to the physical environment using traditional controls (e.g.joysticks, gestures, mouse movement, keyboard buttons, etc.) and anexample navigation method described herein maintains synchronizationbetween the physical world and the virtual scene as the user usesnatural movement to navigate the scene.

Another example for controlling location of movement of the virtualcamera is to use samples from the accelerometer and/or gyroscope sensorsof the user device to calculate changes in the position of the userdevice (without the use of any signals originally transmitted externallyof the user device and received by the user device in some embodiments).

In one example, the change in position or velocity of the user device in3-dimensional global space, v, is related to its acceleration, a, byequation 12.V=∫adt.  (12)

The device's position, p, is related to its velocity by equation 13.p=∫vdt.  (13)

Equations 12 and 13 can be used to modify the velocity and position ofthe virtual camera based on updates from the device's accelerometersensors. However, updates of accelerometer data are received as discretesamples, so discrete alternatives to these equations are used in oneembodiment. The velocity and position at sample k can be found byequations 14 and 15 where ti is the timestamp of sample i.v _(k)=Σ_(i=1) ^(k)(t _(i) −t _(i−1))a _(i)  (14)p _(k)=Σ_(i=1) ^(k)(t _(i) −t _(i−1))ν_(i)  (15)

Equations 14 and 15 are discrete approximations of continuous functionsand numerical discretization error can be reduced by using thetrapezoidal rule or other forms of quadrature. Doing this producesequations 16 and 17.

$\begin{matrix}{v_{k} = {\sum\limits_{i = 1}^{k}\;{\left( {t_{i} - t_{i - 1}} \right)\frac{\left( {a_{i - 1} + a_{i}} \right)}{2}}}} & (16) \\{p_{k} = {\sum\limits_{i = 1}^{k}\;{\left( {t_{i} - t_{i - 1}} \right)\frac{\left( {v_{i - 1} + v_{i}} \right)}{2}}}} & (17)\end{matrix}$

Equations 16 and 17 help reduce error due to discretization in thedescribed embodiment.

Furthermore, the method may also force the velocity to zero if theabsolute value of the acceleration is below a threshold value for someperiod of time to assist with reduction of roundoff errors and errorsresulting from the accelerometer sensors.

Other methods of controlling translation of the virtual camera withinthe virtual model detect when the user takes a footstep with the userdevice and uses the detected footstep to update the view point of thevirtual camera in response to the detection. In one embodiment, datafrom the accelerometer and gyroscope sensors of the user device aremonitored to detect the presence of a footstep.

In one embodiment, detection of a footstep is used to control thevirtual camera to move in a translation direction defined by theorientation of the user device in the physical world at a particularspeed and a constant elevation within the virtual model. In oneembodiment, footsteps are detected by analyzing the periodic nature ofthe output of the accelerometer sensors which are generated as a resultof a user stepping with the user device (e.g., holding the device infront, wearing the device, etc.).

One implementation of this method uses two independent loops which areexecuted by the processing circuitry: an accelerometer sensor samplingloop and a frame update loop which operate at different frequencies. Forexample, a sampling rate (e.g., 60 Hz) is selected and used to samplethe accelerometer data which is generated by the accelerometer sensorsin order to adequately process the output of the accelerometer sensors.The update loop is set to a variable refresh rate which is less than thesampling rate of the accelerometer sensors (e.g., within a range of20-40 Hz) since increased computation is performed during the frameupdate loop and the human eye can perceive motion at a refresh ratesignificantly lower than 60 Hz.

Referring to FIG. 7, an example method of detecting the occurrence offootsteps of a user holding user device and which may be executed byprocessing circuitry of the user device is shown according to oneembodiment. Other methods are possible including more, less and/oralternative acts.

The illustrated example method includes a plurality of independent loopsincluding the accelerometer sensor sampling loop L1 and a frame updateloop L2 which may be executed in parallel.

Initially, the user device is notified at an act A20, that theaccelerometer sensors should be sampled to generate accelerometer data.

At an act A22, the samples of the acceleration data are obtained andstored in storage circuitry.

At an act A24, for each accelerometer sample, the acceleration data istransformed from device coordinates to global coordinates as discussedherein. In one embodiment, only accelerometer data in the global Zdirection is used subsequently below for step detection and theaccelerometer data in the global X and Y directions is not used.

At an act A26, it is determined whether a sufficient number of samplesof the accelerometer data have been obtained to allow filtering (e.g.,sixteen samples).

If not, the loop L1 returns to act A22 to record additional samples ofthe acceleration data.

If so, the loop L1 proceeds to an act A28 to perform filtering of thesamples of the acceleration data to remove noise and other unwantedfrequencies.

Unfortunately, raw accelerometer data obtained from the accelerometersensors is not well-suited for step detection since the data may benoisy even when the device is at rest, and further the signal producedby natural stepping motion is periodic but also contains minor peaks andvalleys that make step detection difficult. Accordingly, one processingmethod filters the output of the accelerometer sensors to removeundesired frequency elements.

In one embodiment described below, filtering of the accelerometer datais provided about an assumed typical stepping frequency at which peopletypically walk (e.g., 1.667 Hz with an associated period ofapproximately 0.6 seconds) in one embodiment. As described below in oneembodiment, a two-step filtering process is used which removes allcomponents except those within a desired frequency range ofapproximately 1 Hz-4 Hz about the typical stepping frequency and whichcorresponds to a typical stepping frequency range of a user. Data withinthe typical stepping frequency range is used to detect a footstep of anindividual in the illustrated example. The discussion proceeds withrespect to an embodiment where low pass filtering is first performed,and thereafter, individual cycles of the accelerometer data are filteredto determine whether any of the individual cycles correspond to a stepof a user.

In one example, digital signal processing (DSP) techniques can beemployed to create a filter that removes unwanted frequencies above thetypical stepping frequency range. For example, one implementationemploys a low-pass filter to remove frequencies above the typicalstepping frequency range. In one embodiment, the low-pass filter has acutoff frequency of 4 Hz to remove the higher frequency components.

In one more specific example, a digital Finite Impulse Response (FIR)filter is utilized to perform the low-pass filtering and may employ a16-element coefficient vector which produces a 16 sample shift in thephase of the filtered signal.

At an example accelerometer sensor sample rate of 60 Hz, this phaseshift means that the filtered signal lags the raw signal by 16/60=0.267seconds. With an FIR filter, a larger number of filter coefficientsusually results in improved filter characteristics. However, a largernumber of filter coefficients produces a longer time lag in the signalunless the sample frequency is also increased. A longer time lag isundesirable because the user begins to notice that steps registered bythe method are out of phase with the actual steps. The user experiencewould be negative if, for example, the user had to wait seconds aftertaking a step to see the scene of the virtual model update. Increasingthe sample rate reduces the lag within the filtered signal, but leavesless time for calculations to be performed between samples. In oneembodiment, a 16-element coefficient vector filter coupled with a 60 Hzrefresh rate represents a reasonable compromise between accuracy incomputation and reaction time of the method based upon current hardwareavailable in typical user devices.

In addition, other filters may be used in other embodiments. In theabove example, a FIR filter was chosen over an Infinite Impulse Response(IIR) filter because a signal passed through a digital IIR filter maycontinue to show oscillations long after the raw signal has stoppedoscillating. In step detection, it is desirable to detect a single stepand ensure that additional ‘false’ steps aren't created by the filter.An FIR filter is more computationally intensive than an IIR filter butdoes not suffer from the ‘ringing’ limitation. Accordingly, theprocessing described according to one embodiment accurately detects thepresence of one footstep of a user.

Referring to FIG. 8, a graphical representation of a raw accelerationdata signal 100 in the global Z axis direction is shown as captured fromthe output of the acceleration sensors and transformed to the globalreference as described herein. The illustrated filtered output signal102 results from the example filtering using the FIR filter describedabove.

At an act A30, the signal resulting from the filtering is examined todetermine whether a step has occurred. The raw acceleration data signalis analyzed to determine if the user is stepping after the filtering ofthe data in the described implementation.

In one embodiment described below, the processing of act A30 analyzesindividual cycles to determine whether any of the individual cyclescorresponds to a footstep of a user stepping with the user device.

Referring to FIG. 9, one method of detecting whether a step has occurredusing the filtered output signal 102 is described in one embodiment. Themethod of FIG. 9 performs additional filtering of the output signal andeffectively functions as a high-pass filter such that the filteredoutput signal 102 comprises only components above approximately 1 Hz inone embodiment. This additional filtering of act A30 in combination withthe filtering of the raw data signal described above of act A28 performsa two-step band pass filtering process which removes all componentsexcept those within the typical stepping frequency range ofapproximately 1 Hz-4 Hz.

In one embodiment, a period of an individual cycle of the accelerometerdata is analyzed with respect to a period of a typical footstep cycle(0.6 seconds), and a cycle is identified as corresponding to a footstepof a user with the user device if the period of the cycle of data isless than the period of a typical footstep cycle. Cycles ofaccelerometer data having periods greater than the period of the typicalfootstep cycle are filtered and not indicated as corresponding to afootstep of the user with the user device. As described below, less thanan entirety of the individual cycle of accelerometer data may beanalyzed to determine whether it corresponds to a footstep by selectionof a cutoff period of time (t_(cutoff)) 106.

The described example method makes use of a negative minimumacceleration threshold (−a_(thresholdmin)) 104, a positive minimumacceleration threshold (+a_(thresholdmin)) 108, and a cutoff period oftime (t_(cutoff)) 106 to determine whether a step has occurred. Thecutoff period of time 106 is set to be longer than half the period of atypical footstep cycle of a user and less than the period of the typicalfootstep cycle of the user. For example, the cutoff period of time 106is 0.5 seconds in one embodiment. Use of different cutoff periods oftime 106 will adjust the frequencies which are passed by the high-passfiltering in the described example.

Each (filtered) acceleration sample in the global Z direction of signal102 is examined to determine whether the signal has ‘crossed’+/−a_(thresholdmin) 104, 108. If −a_(thresholdmin) 104 has been crossedand the signal 102 is decreasing, the moment in time at which thecrossing occurred is recorded as t_(lower) 103. Subsequent samples areexamined to determine whether +a_(thresholdmin) 108 has been crossedwhile the signal 102 is increasing. A step is detected if signal 102crosses −a_(thresholdmin) 104 while decreasing and crosses+a_(thresholdmin) 108 while increasing within t_(cutoff) 106 seconds tobe registered in the described example.

The above describes analysis of one possible portion of a cycle of thedata with respect to the cutoff period of time 106. There are othercrossings of thresholds 104, 108 which may be analyzed in otherembodiments. If the crossings of thresholds 104, 108 by signal 102 arenumbered 1-8 from left to right, portions of the signal 102 between thefirst and third crossings, between the second and fourth crossings, orbetween the fourth and sixth crossings may be analyzed with respect tothe cutoff period of time 106 to determine whether an individual cyclecorresponds to a footstep (i.e., in addition to the portion of thesignal 102 which corresponds to the third and fifth crossings describedabove with respect to crossing 103 and the cutoff period of time 106shown in FIG. 9).

Accordingly, in one embodiment, only a portion of an individual cycle ofaccelerometer data is compared with a predefined length of timecorresponding to the cutoff period of time 106 to determine whether theindividual cycle corresponds to a footstep.

The example of FIG. 9 is described with respect to analysis of thesignal 102 while increasing with respect to +a_(thresholdmin) 108 todetect the presence of a footstep. In another embodiment, the signal 102may be analyzed in the same manner while decreasing with respect to−a_(thresholdmin) 104 to detect the presence of a footstep within thecutoff period of time. In this example, the cutoff period of time may beinitiated when the signal 102 crosses +a_(thresholdmin) 108.

The second step of filtering of act A30 filters cycles havingfrequencies less than the typical stepping frequency range (e.g., below1 Hz) in one implementation. This filtering of FIG. 9 helps to ensurethat lower frequency accelerometer oscillations due to device rotationare not mistaken for footsteps. More specifically, the above-describedexample detection operates as a high-pass filter upon the low passfiltered output of the FIR filter in the example two-step filteringprocess and footsteps are detected if the accelerometer data includesfrequencies within the typical stepping frequency range (e.g., 1-4 Hz)in one embodiment. A notification that a step has been detected may begenerated to indicate the presence of the footstep of the user steppingwith the user device as a result of the detection of such frequencies.

The two-step process of first applying a digital low-pass filter to thesignal and then applying the time cutoff technique mentioned aboveaccording to one embodiment operates as a band-pass filter. The digitallow pass filter removes frequencies above a certain range (e.g., 4 Hz),and then the time cutoff technique through use of the cutoff period oftime removes frequencies below a certain cutoff frequency (e.g., 1 Hz).

This two-step filtering arrangement allows for a very narrow band offrequencies to be filtered while maintaining computational efficiencycompared with a traditional digital band pass filter which focuses in ona narrow frequency band and which uses a higher sample rate and/or alarger coefficient vector. However, increasing the sample rate decreasesthe amount of time available to make computations between samples andincreasing the size of the coefficient vector increases the number ofcomputations that must be performed for each sample. The two-stepfiltering process according to one embodiment of the disclosure isefficient which enables use in real-time as a user is stepping.

The two-step filtering process allows the lower cutoff frequency to havean infinitely small transition zone. In particular, signals below thecutoff frequency are completely attenuated and signals above the cutofffrequency are not attenuated at all in the described example arrangementwhich results in a faster filter of increased purity compared with sometraditional digital filter design techniques which provide relativelywide transition zones with signal distortion introduced at and aroundthe cutoff frequencies.

The use of the cutoff period of time for the lower frequency cutoffresults in the low frequency cutoff being extremely sharp and veryaccurate which is applicable to narrow range band pass filtering.Additionally, this example technique yields an output signal where thelower end of the pass band signal suffers less from ripples or ringingthat are often associated with traditional digital filtering techniques.

Not all oscillations that fit within the time window described above areregistered as footsteps in one embodiment. Stepping is characterized byrelatively small accelerations associated with small displacements ofthe user device. If large accelerations are detected, experience hasshown that these accelerations are usually associated with largedisplacements of the user device and usually aren't due to foot stepsbeing taken. Instead, these large accelerations typically occur when theuser is moving the user device to look down at the floor or up at theceiling of the virtual model.

In one embodiment, a maximum acceleration threshold (a_(thresholdmax))110 is used to ensure that accelerations with a relatively largemagnitude are not registered as footsteps. This thresholda_(thresholdmax) 110 was empirically derived as −0.285 g in oneimplementation (threshold 110 is not drawn to scale in FIG. 9).Furthermore, an amplitude (a_(amplitude)) 105 of each cycle is found byrecording the minimum acceleration since the signal last crossed 0.0 gwhile decreasing and prior to increasing. When a footstep is detected asabove, a_(amplitude) is compared with a_(thresholdmax) and a footstep isregistered as being detected only if the absolute value of a_(amplitude)is less than the absolute value of a_(thresholdmax) in this embodiment.Otherwise the detection is disregarded as being a relatively largeamplitude not due to a footstep.

Accordingly, in one embodiment, the detected presence of a footstep isnot indicated as being a detected footstep if the accelerometer data hasan amplitude which is larger than a threshold amplitude(a_(thresholdmax)) which is indicative of a typical footstep. In oneembodiment, an amplitude of a cycle of the accelerometer data isdetermined and compared with a maximum acceleration threshold, and thedetected footstep is only indicated as being detected if the amplitudeof the accelerometer data is less than the maximum accelerationthreshold.

A user with a very light step produces acceleration data in the global Zdirection which has a smaller amplitude compared with a user with aheavy step. Likewise, a user stepping on cement will produce differentacceleration data than the same user stepping on grass. One embodimentof the footstep detection method adapts to these differences and detectssteps under a wide variety of conditions.

In one implementation, the threshold acceleration, a_(thresholdmin) isadjusted and adapted to the acceleration data being generated based onthe amplitude of the filtered signal. In particular, the thresholdacceleration a_(thresholdmin) is set according to equation 18:a _(thresholdmin)=min(a _(minimum) ,C·a _(amplitude))  (18)where a_(minimum) is a predetermined minimum acceleration value,a_(amplitude) is the amplitude of the accelerometer data of the mostrecent cycle and corresponding to the footstep of a user, and C is ascalar that sets a_(thresholdmin) to some percentage of a_(amplitude).The values of a_(minimum) and C may be empirically derived and hardcodedor set by the user and examples of these values are a_(minimum)=0.025 gand C=0.25 in one implementation. The value of a_(amplitude) is largerfor a user with heavy steps as detected by the accelerometer sensors ofthe user device compared with a user with light steps. Adaptinga_(thresholdmin) in this way allows the algorithm to adjust to changingconditions (i.e., how the user steps and upon different surfaces).

Referring again to FIG. 7, acts A28 and A30 operate as a two-stepfiltering process of the accelerometer data in one embodiment.

If the processing of the acceleration data does not result in a stepbeing detected at act A30, the process returns to act A22.

However, if the processing of the acceleration data results in a stepbeing detected at act A30, the process proceeds to an act A32 to publisha step notification which indicates that a footstep has been detected.

The discussion below proceeds to virtual model navigation. However, theabove-described example two-step filtering process may also be appliedor utilized in other applications for other filtering purposes in otherembodiments. In particular, the two-step process of filtering to createa band-pass filter may be useful in domains outside of step detectionand device motion. The two-step filter may be used in situations whereit is useful to filter all but a narrow band of frequencies, where thespeed of computing and analyzing the filtered data is important (e.g.processing a signal in real time), when it is desirable to clearlydistinguish between included and excluded frequencies that are close tothe lower frequency cutoff, and/or when it is desirable to remove‘ringing’ in the filtered low frequencies. Some more specificillustrative additional applications include sonar and radar processing,seismic data processing, audio and speech processing, image processing,etc.

As mentioned above, the frame update loop L2 may be executed in parallelwith the loop L1. At an act A40 of loop L2, the process attempts toaccess a published step notification indicating that a user footstep hasbeen detected.

At an act A42, it is determined whether a step notification indicatingthe detection of a footstep has been published by loop L1.

If a step notification has been published, the method proceeds to an actA44 to record the time of the detection of the footstep.

Thereafter, or if the condition of act A42 is negative, the methodproceeds to an act A46 to calculate a camera move distance which may beused to reposition the virtual camera at a new view point within thevirtual scene as a result of a step being detected.

In particular, with step notifications being sent to the frame updateloop L2 from loop L1, it is possible to calculate the distance thevirtual camera should move for each frame update of the virtual scene.In one embodiment, this distance d is calculated by Equation 19:d=t _(update) ·f _(step) ·l _(step) ·A  (19)where t_(update) is the time since the last frame update, f_(step) isthe user's step frequency, l_(step) is the length of the user's step,and A is an attenuation function.

To calculate t_(update), a timestamp is recorded at each frame and thetimestamp for the last frame is subtracted from the timestamp for thecurrent frame. The step frequency f_(step) is assumed to be a constantor may alternatively be calculated using accelerometer data in exampleembodiments. As mentioned above, an assumed step frequency f_(step) atwhich people typically walk is 1.667 Hz (having a period ofapproximately 0.6 seconds) in one embodiment.

The step length, l_(step), is set to a reasonable default value (e.g.,30 inches) and is modifiable by the user. In another example, l_(step)can be obtained by instructing the user to walk a measured distanceholding the user device while the algorithm picks up steps and dividingthe distance covered by the number of steps provides the step lengthl_(step). With location sensors capable of measuring position ordistance traveled with high accuracy, l_(step) is calculated andadjusted at run time using the location sensors in one embodiment.

In one embodiment, the attenuation function A can be computed accordingto Equation 20:

$\begin{matrix}{A = {\max\left( {0,{1 - \frac{t_{current} - t_{step}}{t_{live}}}} \right)}} & (20)\end{matrix}$where t_(current) is t the timestamp of the current frame, t_(step) isthe timestamp of the last frame for which a step was recorded, andt_(live) t is the ‘time to live’ for a step. In one example, t_(live) tis a constant of 0.7 seconds. Thus, the attenuation A is 1.0 for theframe when a step is first recorded, and linearly decreases to 0.0 foreach frame drawn after the step. The use of the attenuation function Ain the described example ensures that the virtual camera moves while theuser is stepping and that the virtual camera stops moving if a step hasnot occurred after a time of t_(live).

Accordingly, in some embodiments, the method detects a small number ofsteps (even a single step) and responds to it in real time or in amanner that feels like real time to the user. In particular, the virtualcamera is moved within the virtual scene following the detection of asingle footstep by a user. Use of equation 20 provides a good compromisewhere the virtual camera moves instantly when a single step is detectedand it causes the virtual camera's velocity to taper linearly (notasymptotically) toward 0 as time elapses without another step. In oneembodiment, translation movement of the virtual camera is initiatedresponsive to the detection of an initial single footstep after the userdevice has been stationary (i.e., without any detected footsteps) andprior to detection or indication of another/subsequent footstep.Equation 20 also produces a virtual camera velocity in the virtual scenethat is proportional to the step frequency f_(step) since the averageattenuation will be higher for higher step frequencies.

Referring to FIG. 10, an attenuation function 112 is shown when stepsoccur every 0.3 seconds and t_(cutoff) is 0.9 seconds. An average 114 ofattenuation function 112 is also depicted.

Referring to FIG. 11, an attenuation function 112 a is shown when stepsoccur every 0.6 seconds and t_(cutoff) is 0.9 seconds. An average 114 aof attenuation function 112 a is also depicted.

As shown in FIGS. 10-11, the use of equation 20 produces higher averageattenuations for higher step frequencies which results in largerdistances being determined in equation 19 providing larger displacementsof the virtual camera (and increased velocity of the virtual camera)compared with lower step frequencies.

Accordingly, in one embodiment, frame updates occur in a separate loopL2 from the accelerometer sensor sampling loop L1, and each time throughloop L2, the method checks to see if a step notification has been sentfrom the accelerometer sampling loop L1.

If a step notification is received, the timestamp for the frame on whichthe notification was received is recorded as t_(step). Then, thedistance to move the virtual camera is computed for all frames in thedescribed example, regardless of whether a step notification isreceived. The camera move distance could be (and often is) zero sincethe camera move distance is a function of the time since the lastrecorded step. This quantity decreases linearly from the time a step isrecorded until the camera move distance reaches 0.

Following the calculation of the camera move distance, the methodproceeds to an act A48 to set the position of the virtual camera at anupdated view point within the virtual model based on the computations ofact A46. In one embodiment, the distance between the first and secondview points is defined by equation 19.

In one embodiment, the virtual camera is moved the camera move distancein the positive view direction of the virtual camera which may bedefined by the view direction of the user device 10 and which may resultin changes of elevation of the view point or remain at a constantelevation as a previous view point as described in illustrativeembodiments above.

At an act A50, the virtual scene is redrawn with the virtual cameralocated at the updated view point within the virtual scene.

The following describes control of the virtual camera within the virtualscene during navigation of a virtual model according to exampleembodiments herein. In one embodiment, the virtual camera has 6 degreesof freedom: 3 rotational and 3 translational and the virtual camera canbe rotated about its X, Y, or Z axes, and it can be translated along itsX, Y, or Z axes of the scene coordinate reference frame. The degrees offreedom of the virtual camera are controlled or manipulated by variouscombinations of traditional controls (joysticks, gestures, mousemovements, key presses, etc.) and natural movements (stepping, devicerotation, etc.) in illustrative embodiments described herein.

For illustration purposes, the possible configuration for control of thevirtual camera may be simplified to three different groupings of degreesof freedom: rotation about all 3 axes (R(x,y,z)), translationconstrained to a plane (T(x,y)) wherein the virtual camera is positionedat a constant elevation in a scene, and translation along all 3coordinate axes (T(x,y,z)). The controls may also be grouped into threecategories: traditional (joysticks, gestures, mouse movements, keypresses, etc.), natural translation (stepping or otherwise moving in thephysical world), and natural rotation (rotating the user device as if itwere the virtual camera). In some embodiments, R(x,y,z), T(x,y), and/orT(x,y,z) are controlled through traditional controls or R(x,y,z) iscontrolled through natural rotation of the user device, In anotherexample embodiment, T(x,y) or T(x,y,z) are controlled through naturaltranslation and any other method of controlling R(x,y,z). In anotherembodiment, R(x,y,z) is controlled using natural rotation of the userdevice combined with any control of translation. In yet anotherembodiment, natural rotation of the user device is used to controlR(x,y,z) and traditional methods (e.g., joysticks) are used to controlT(x,y,z) to produce the sensation of “flying” through the virtual scene.In another embodiment, T(x,y,z) is controlled using natural translationof the user device and the some angle given with respect to the XY plane(controlled through natural rotation or traditional controls) to producea sense of the user growing or shrinking as the user navigates themodel.

As described below in some embodiments, processing circuitry isconfigured to combine and integrate content from the physical world withcontent of a virtual scene to create views of the virtual model whichalso include content of the physical world. In one example, images(e.g., video of physical world content) generated by the camera 38 ofthe user device may be combined and integrated with virtual contentwithin a virtual scene of the virtual model.

In one example discussed further below, video of the physical world maybe shown within a translucent surface such as a window of the virtualmodel if the virtual camera is oriented to look through the surface in agiven view (e.g., physical world content from the camera of the userdevice may be shown through a window of a virtual scene of the virtualmodel in the form of an architectural structure).

In one example for navigating virtual models in the form ofarchitectural structures, the video camera image could be used and shownas the backdrop for the virtual scene and the virtual content of thevirtual model may be overlaid or drawn on top of the backdrop andportions of the backdrop (i.e., physical world content in this example)are visible through translucent surfaces of the virtual content.Accordingly, when a view of the virtual model includes a translucentsurface of virtual content (e.g., a virtual window), then image datafrom the camera 38 is visible through the window as the backdrop.

In some embodiments, the pixels which correspond to the translucentsurfaces in a view of the virtual model are identified and image datafrom respective pixels of the physical world content generated by thecamera 38 are visible at the identified pixels (e.g., if a pixel at alocation x, y of the view of the virtual model is translucent, then thecontent of the pixel at the corresponding location x, y of the imagedata is visible at the location x, y of the view of the virtual model).

In some embodiments, the translucent surfaces may be transparent wherethe corresponding pixels of image data may be shown in the view withoutmodification. In other embodiments, the pixels of image data may bealtered to indicate the translucent nature of the surface.

In one embodiment, image data obtained from the camera of the userdevice is only shown at translucent surfaces at an outermost boundary orperiphery of the virtual model. For example, the processing circuitrymay determine that a translucent surface which is present in a givenview is located at the outermost boundary of the virtual model, and inresponse thereto, control the image data of the physical world to bevisible within the translucent surface in the view. Furthermore, virtualcontent of the virtual model (as opposed to image data from the cameraof the user device) would be visible through translucent surfaces whichare located within the interior of the virtual model (i.e., not at theperiphery).

These described example embodiments provide the user with a perceptionof being located within the virtual model and seeing the physical worldcontent through translucent surfaces at the periphery of the virtualmodel. Views of the virtual model as observed from view points which areinside of the virtual model are created and content of the physicalworld which is outside of the virtual model is visible in the createdviews of the virtual model in one embodiment. As mentioned above,virtual models of other subject matter may be navigated in otherembodiments (e.g., a user may peer out of the eyes of a virtual model ofa living organism in another illustrative example).

In one embodiment, the Unity graphics engine may be utilized to combineand integrate image content from the camera of the user device into aview of a virtual model. In one specific implementation, a backgroundcamera object is created using the graphics engine and severalcomponents are attached to this object. First, a script component isattached to the background camera object to allow custom implementationsof the ‘Start’ and ‘Update’ functions with the Unity graphics engine.Second, a camera component is added to the background camera objectwhich encapsulates information necessary to correctly render an imagefrom the video camera 38 to the display screen.

For example, when the camera component is created, its ‘depth’ parameteris set to −1. This value is intended to be lower than the depth valuesfor any other camera component in the scene which causes the pixelsdefined by the background camera object to be rendered to the screendisplay before any other camera object, effectively making thebackground camera object responsible for drawing the background image(the image that will appear behind all others) for the virtual scene,and accordingly, portions of the background image are visible throughtranslucent surfaces present in a given view of the virtual model.

In addition, the ‘Start’ function of the background camera object in theUnity graphics engine contains code to iterate over all of the videocameras associated with the user device. Once a camera of the userdevice that is designated a rear facing camera (i.e., view direction ofthe camera which is the same as the view direction of the user device)is located by the graphics engine, the camera is turned on and thegraphics engine stores a handle to it.

As mentioned above, the user device 10 defines a view direction which isnormal to the plane of the display screen in one embodiment. The userdevice 10 may contain a plurality of cameras having view directions indifferent directions from the user device 10 (e.g., present in differentsides of the user device). In one embodiment, a camera 38 of the userdevice which has a view direction in the same direction as the viewdirection of the user device is used to generate the physical worldcontent for inclusion in the views of the virtual model.

The video image from the camera of the user device is then connected toa GUITexture object in the Unity graphics engine in one embodiment. AGUITexture is an image that always resides in the 2-dimensional plane ofthe screen. The GUITexture is associated with the background cameraobject which causes the background camera object to render image datafrom camera 38 of the user device to the display screen during frameupdates.

The ‘Start’ function of the background camera object also contains logicto orient the image of camera 38 of the user device correctly based oncurrent orientation of the display screen of the user device. Forexample, if an iOS based user device is oriented so that it is in‘landscape’ orientation (as opposed to ‘portrait’ orientation) with the‘Home’ button on the right, the image of the camera of the user devicemay be used without modification. If, however, the device is inlandscape orientation with the ‘Home’ button on the left, the image ofthe camera of the user device is inverted so that it does not appearupside down. This orientation check and correction is performed at frameupdates to account for device rotation between frames in oneimplementation.

It is desirable to fill the entire display screen 32 with pixels fromthe camera 38 as the backdrop and virtual content may be drawn over thebackdrop in the desired embodiment However, the resolution of the camera38 of the user device doesn't necessarily match the resolution of thescreen 32. The ‘Update’ function for the background camera object in theUnity graphics engine accommodates these differences.

If both axes of the camera of the user device contain the same number ormore pixels than the display screen, the video image is simply clipped(cropped) to match the resolution of the screen. The Unity graphicsengine performs this operation automatically as part of the renderingprocess. If the video image is smaller than the screen in eitherdimension, the video image is scaled so that it fills the display screenin one implementation. A scale factor is computed in the ‘Update’function of the Unity graphics engine and the Unity graphics engineperforms the necessary image processing operations to scale the image tofit the display screen in one implementation.

Displaying the feed from the camera 38 according to the above exampleimplementation ensures that the image from the camera 38 is always shownas the backdrop for the scene. This produces the illusion that the useris looking through the view of the virtual model at the backdrop.

Referring to FIG. 12, a virtual scene of an example virtual model 120 inthe form of a simple architectural structure 122 is shown. In thedepicted example, the virtual scene includes a virtual camera which isused to generate different views of the virtual model depending upon theview point and view direction of the virtual camera and the generatedviews may be displayed using the display screen of the user device asshown in FIG. 13.

In FIG. 12, virtual camera is shown at an initial location (i.e., viewpoint) 124 within the virtual scene pointing in a view direction 126within the virtual scene. In this example, the architectural structure122 includes a translucent window 128. In this embodiment, image datafrom the camera of the user device (i.e., content of the physical world)is used to form a backdrop 130 within the virtual scene and portions ofthe backdrop 130 may be visible through the window 128 in differentviews which are generated of the virtual model.

For example, processing circuitry may access model data of the virtualmodel from appropriate storage circuitry to draw a view of the virtualmodel for a specified view point 124 and view direction 126 of thevirtual camera within the virtual scene. In one embodiment, content ofthe physical world in the backdrop 130 may be shown through atranslucent surface (e.g., window 128) overlaid upon the backdrop 130within the view to be drawn. As mentioned above, camera 38 may generatevideo images of the physical world content, and the pixels of thebackdrop 130 which correspond to the translucent surface within the viewbeing drawn may change between different frames. In one embodiment,image data of the physical world content may be retrieved from videoframes which correspond in real time such that current image data isshown in the view of the virtual model.

The Unity graphics engine combines and integrates the image data fromthe camera 38 of the user device into views of the virtual model in thisexample by showing respective portions of the image data content whichwould be visible through the window 128 corresponding to locations andview directions of the virtual camera. If the window 128 is within aview of the virtual model according to a location and view direction ofthe virtual camera at a given moment in time, then image data content ofthe backdrop 130 which corresponds to portions of the backdrop 130 whichare visible through the window 128 given the location and view directionof the virtual camera at the given moment in time are combined andintegrated with content of the virtual model to be visible through thewindow 128 while the remaining portions of the virtual scene wouldinclude content of the wall 132 of the architectural structure 122.

As described above, different inputs (e.g., via user controls or sensorsof the user device) may be used to change one or more of the view pointand/or view direction of the virtual camera between views of the virtualmodel. In the illustrated example, the virtual camera has been moved toa new location 124 a and points in a different view direction 126 a at asubsequent moment in time compared with initial location 124 and initialview direction 126. Different views of the virtual model are generatedand displayed using the display screen 32 corresponding to the differentview points 124, 124 a and view directions 126, 126 a. In addition,different portions of the backdrop 130 would be shown in the differentviews generated by the virtual camera at the different view points 124,124 a and pointing in the different view directions 126, 126 a.

Referring to FIG. 13, a view of the virtual model which may be generatedby the virtual camera positioned and oriented in accordance with one ofthe examples shown in FIG. 12 is displayed on the display screen 32 ofthe user device. The virtual scene includes virtual content (i.e., thewall 132 of the virtual model which overlies respective portions of thebackdrop) and content of the physical world in the form of the imagedata utilized as the backdrop 130 and which is visible through atranslucent surface of the virtual model (i.e., window 128).

In one embodiment, the field of view (FOV) of the virtual camera isconfigured to correspond to the field of view of the camera 38 of theuser device to create a realistic experience for the user when imagedata of the user camera s included in a virtual scene. Otherwise, theuser experience may be negatively impacted.

For example, if the virtual camera's FOV is larger than the FOV of thecamera of the user device, the image data of the backdrop will appeartoo close in the virtual scene. In one more specific example embodiment,FOV parameters of the virtual camera are set to be the same as thecorresponding FOV parameters of the camera 38 of the user device.

The field of view (FOV) parameters of the camera 38 are typically fixedby the hardware and designed to match the aspect ratio of the displayscreen 32 of the user device in a particular orientation. In oneembodiment, the field of view parameters to be utilized for the virtualcamera may be specified in the Unity graphics engine.

The FOV parameters may be different on various types of cameras used inuser devices. Accordingly, in at least one operational method,information of these FOV parameters of the camera 38 of the user devicebeing utilized may be obtained at run time in different ways.

In example implementations, the FOV parameters of the camera 38 of theuser device can be obtained by querying the API of the user device,reading the device's hardware specifications, experimentation, orreading a data table of parameters for different video cameras or userdevices stored within the storage circuitry of the user device. Once theFOV parameters of the hardware configuration of camera 38 are obtained,they may be used to configure the field of view of the virtual camera tomatch the field of view of camera 38 being used by the user device 10.Images produced from both cameras (virtual camera and camera 38) can beintegrated seamlessly once the virtual camera's FOV parameters arematched to the camera 38 of the user device.

In compliance with the statute, the invention has been described inlanguage more or less specific as to structural and methodical features.It is to be understood, however, that the invention is not limited tothe specific features shown and described, since the means hereindisclosed comprise preferred forms of putting the invention into effect.The invention is, therefore, claimed in any of its forms ormodifications within the proper scope of the appended aspectsappropriately interpreted in accordance with the doctrine ofequivalents.

Further, aspects herein have been presented for guidance in constructionand/or operation of illustrative embodiments of the disclosure.Applicant(s) hereof consider these described illustrative embodiments toalso include, disclose and describe further inventive aspects inaddition to those explicitly disclosed. For example, the additionalinventive aspects may include less, more and/or alternative featuresthan those described in the illustrative embodiments. In more specificexamples, Applicants consider the disclosure to include, disclose anddescribe methods which include less, more and/or alternative steps thanthose methods explicitly disclosed as well as apparatus which includesless, more and/or alternative structure than the explicitly disclosedstructure.

What is claimed is:
 1. A virtual model viewing method comprising: using a user device, accessing model data which may be used to generate a plurality of views of a virtual model which comprises a structure; using a camera of the user device, generating image data of content of the physical world; using the model data and the image data, creating one of the views of the virtual model which comprises at least some of the content of the physical world; displaying the one view of the virtual model using a display screen of the user device; wherein the creating the one view of the virtual model comprises creating using a virtual camera positioned at a view point which is located within an interior of the structure of the virtual model; wherein the displayed one view of the virtual model includes an interior portion of the structure of the virtual model; and wherein the at least some of the content of the physical world is visible within an area of the displayed one view of the virtual model which is defined by the interior portion of the structure of the virtual model.
 2. The method of claim 1 wherein the camera of the user device has a field of view, and further comprising configuring the virtual camera to have a field of view which corresponds to a field of view of the camera of the user device.
 3. The method of claim 1 further comprising: obtaining information regarding the field of view of the camera of the user device; and configuring a field of view of the virtual camera using the obtained information.
 4. The method of claim 3 wherein the configuring comprises configuring the virtual camera to have a field of view which corresponds to the field of view of the camera of the user device.
 5. The method of claim 4 wherein the configuring comprises configuring the virtual camera to have a field of view which is the same as the field of view of the camera of the user device.
 6. The method of claim 4 wherein the obtaining comprises obtaining the information regarding the field of view from the user device.
 7. The method of claim 1 further comprising controlling a view direction of the virtual camera using a view direction of the user device in the physical world.
 8. The method of claim 7 wherein the controlling comprises controlling the view direction of the virtual camera to correspond to the view direction of the user device.
 9. The method of claim 7 wherein the view direction of the user device is in a direction which is orthogonal to a plane which includes the display screen of the user device.
 10. The method of claim 9 wherein the view direction of the user device is the same as a view direction of the camera of the user device.
 11. The method of claim 10 further comprising aligning a scene coordinate reference frame of the virtual model with a global coordinate reference frame of the physical world.
 12. The method of claim 11 wherein the controlling comprises controlling the view direction of the virtual camera in the scene coordinate reference frame to be the same as the view direction of the user device in the global coordinate reference frame.
 13. The method of claim 11 wherein the aligning comprises aligning using a user input.
 14. The method of claim 1 wherein the area of the displayed one view of the virtual model is a translucent surface.
 15. The method of claim 14 wherein the at least some of the content of the physical world is shown as a backdrop within the one view.
 16. The method of claim 14 wherein the image data is shown as a backdrop within the one view.
 17. The method of claim 14 wherein the at least some of the content of the physical world is visible within the area of the one view as a result of the translucent surface being located at an outermost boundary of the virtual model.
 18. The method of claim 1 wherein the at least some of the content of the physical world which is visible in the area is outside of the virtual model.
 19. The method of claim 1 wherein the structure of the virtual model is an architectural structure and the area corresponds to a window within the interior portion of the structure which is an interior wall.
 20. The method of claim 1 wherein the interior portion of the structure of the virtual model surrounds an entirety of the area which includes the at least some of the content of the physical world.
 21. The method of claim 1 wherein the virtual model includes a translucent surface which corresponds to the area, the content of the physical world is used as a backdrop, and the virtual model is overlaid upon the backdrop.
 22. An article of manufacture comprising non-transitory storage media storing programming instructions configured to cause processing circuitry to perform the method of claim
 1. 23. A computing system configured to perform the method of claim
 1. 